Objectives

1. Understand the logic of statistical testing in clinical trials
2. Demonstrate an understanding of key statistical concepts: hypothesis testing, \(p\) values, confidence intervals and power
3. Be able to describe and interpret common measures of effect used in clinical epidemiology
4. Be able to critically appraise a clinical trial

Clinical epidemiology

Bias

refers to any systematic error in estimating the effect of a drug, exposure or risk factor on a specified outcome

Random error

variation that occurs within stochastic processes

Sources of error

Critical Appraisal

The task of critical appraisal is to appraise the paper and decide whether the findings are of relevance to your practice.

The simplest advice to give about critical appraisal is simply to read the paper and think about it.

Critical Appraisal Tool (Oxford Centre for Evidence-based Medicine)

1. Determine the PICO for the study
2. Appraise the study methods
3. Interpret the results of the study
4. Consider external validity

Determining the PICO for a study

Task

Focusing on the primary endpoint of Voysey et al. (2021):

1. Identify the PICO
2. Identify the null hypothesis
3. Interpret the statistical result

Based on the information available to you, what inference does the study support?

1. Determine the PICO

Primary endpoint and analysis plan

Outcome

There were 30 (0.5%) cases among 5807 participants in the vaccine arm and 101 (1.7%) cases among 5829 participants in the control group, resulting in vaccine efficacy of 70.4% (95.8% CI 54.8–80.6; table 2; figure).

How would you interpret this result?

Understanding key aspects of the statistical inference

2. Appraise the study methods

• Was the assignment of patients to treatments randomized?

Yes

• Were the groups similar at the start of the trial

Yes—at least on a per-trial level. The participants in each trial differ quite a bit (see Table 1).

• Aside from the allocated treatment, were groups treated equally?

Lots of differences (e.g. timing of second dose, dosing, follow-up). That said: methods pre-approved with regulators.

• Were all patients who entered the trial accounted for? And were they analysed in the groups to which they were randomized?

See appendix for CONSORT participant flow diagram. Participants analysed according to vaccines they received (intention-to-treat analysis conducted as a sensitivity analysis)

• Were the measures objective or were the patients and clinicians kept “blind” to which treatment was being received?

Measures seem appropriately objective. The studies differed in terms of masking (single, double blind).

3. Interpret the results

• How large was the treatment effect?

Efficacy: 70.4% (95.8% CI 54.8–80.6)

• How precise was the estimate of the treatment effect?

This is a judgment call. Given this is an interim result and the lower bound of the confidence interval is considerably greater than 20%, it seems reasonable to suggest the result is sufficiently precise.

4. External validity

• Will the results help me in caring for my patient?
• Alternatively: How will I apply the results?

Take homes

• The most-often used cut-off for \(p\) values used in clinical research is 0.05.
• If the \(p\) value is \(< 0.05\), the result will be considered statistically significant
• A statistically significant result for the primary endpoint of a trial is more trustworthy than statistically significant results on secondary endpoints or subgroups—the trial was set up the test the primary endpoint.
• Once you have determined that the primary endpoint of trial was statistically significant, the next question is to determine whether the magnitude of the effect is clinically significant

Measuring the effects of interventions

Relative risk

\[RR = \frac{I_t}{I_c}\]

Relative risk reduction (or increase)

\[ RRR = 1 - RR \]

The absolute risk difference (absolute risk reduction or absolute risk increase)

\[ARR = I_c - I_t \]

\[ARI = I_t - I_c\]

The number needed to treat (\(NNT\)) and/or harm (\(NNH\))

\[ NNT = 100/ARR \]

\[ NNT = 100/ARI \]

Voysey et al. (2021)

\[ I_t = 30/5807 = 0.005 = 0.5\%\]

\[ I_c = 101/5809 = 0.017 = 1.7\% \]

\[RR = \frac{0.5}{1.7} = 0.294\]

\[ RRR = 0.706\]

\[ARR = 1.7 - 0.5 = 1.2\%\]

\[NNT = \frac{100}{1.2} = 83.333\]

Review objectives

1. Understand the logic of statistical testing in clinical trials
2. Demonstrate an understanding of key statistical concepts: hypothesis testing, \(p\) values, confidence intervals and power
3. Be able to describe and interpret common measures of effect used in clinical epidemiology
4. Be able to critically appraise a clinical trial